Risk Bounds for Mixture Density Estimation
نویسندگان
چکیده
In this paper we focus on the problem of estimating a bounded density using a finite combination of densities from a given class. We consider the Maximum Likelihood Procedure (MLE) and the greedy procedure described by Li and Barron [6, 7]. Approximation and estimation bounds are given for the above methods. We extend and improve upon the estimation results of Li and Barron, and in particular prove an O( 1 √ n ) bound on the estimation error which does not depend on the number of densities in the estimated combination. This report describes research done at the Center for Biological& Computational Learning, which is in the Department of Brain & Cognitive Sciences at MIT and which is affiliated with the McGovern Institute of Brain Research and with the Artificial Intelligence Laboratory. This research was sponsored by grants from: Office of Naval Research (DARPA) Contract No. N00014-00-1-0907, Office of Naval Research (DARPA) Contract No. N00014-02-1-0915, National Science Foundation (ITR/IM) Contract No. IIS-0085836, National Science Foundation (ITR/SYS) Contract No. IIS-0112991, National Science Foundation (ITR) Contract No. IIS-0209289, National Science Foundation-NIH (CRCNS) Contract No. EIA0218693, and National Science Foundation-NIH (CRCNS) Contract No. EIA-0218506. Additional support was provided by: AT&T, Central Research Institute of Electric Power Industry, Center for e-Business (MIT), DaimlerChrysler AG, Compaq/Digital Equipment Corporation, Eastman Kodak Company, Honda R&D Co., Ltd., ITRI, Komatsu Ltd., The Eugene McDermott Foundation, Merrill-Lynch, Mitsubishi Corporation, NEC Fund, Nippon Telegraph & Telephone, Oxygen, Siemens Corporate Research, Inc., Sony MOU, Sumitomo Metal Industries, Toyota Motor Corporation, WatchVision Co., Ltd, and the SLOAN Foundation.
منابع مشابه
Risk Bounds for Density Estimation
Motivation: Li and Barron [6, 7] give a greedy method for estimating densities by a finite mixture. The trade-off between approximation and estimation error is present in their risk bound in terms of the optimal number of components used for density estimation. Using techniques from Empirical Process theory [11] and results from geometry in Banach Spaces [5], we improve the bounds and show that...
متن کاملApplications of Empirical Processes in Learning Theory: Algorithmic Stability and Performance Guarantees
First, we demonstrate how the Contraction Lemma for Rademacher averages can be used to obtain tight performance guarantees for learning methods [3]. In particular, we derive risk bounds for a greedy mixture density estimation procedure. We prove that, unlike what is suggested in the literature, the number of terms in the mixture is not a bias-variance trade-off for the performance. Our upper bo...
متن کاملOn Bayes Risk Lower Bounds
This paper provides a general technique for lower bounding the Bayes risk of statistical estimation, applicable to arbitrary loss functions and arbitrary prior distributions. A lower bound on the Bayes risk not only serves as a lower bound on the minimax risk, but also characterizes the fundamental limit of any estimator given the prior knowledge. Our bounds are based on the notion of f -inform...
متن کاملProbability Density Estimation via Infinite Gaussian Mixture Model: Application to Statistical Process Monitoring
The primary goal of multivariate statistical process performance monitoring is to identify deviations from normal operation within a manufacturing process. The basis of the monitoring schemes is historical data that has been collected when the process is running under normal operating conditions. This data is then used to establish confidence bounds to detect the onset of process deviations. In...
متن کاملA Berry-Esseen Type Bound for a Smoothed Version of Grenander Estimator
In various statistical model, such as density estimation and estimation of regression curves or hazard rates, monotonicity constraints can arise naturally. A frequently encountered problem in nonparametric statistics is to estimate a monotone density function f on a compact interval. A known estimator for density function of f under the restriction that f is decreasing, is Grenander estimator, ...
متن کاملMultivariate Density Estimation: a Support Vector Machine Approach
A Support Vector Machine (SVM) algorithm for multivariate density estimation is developed based on regularization principles and bounds on the convergence of empirical distribution functions. The algorithm is compared to Gaussian Mixture Models (GMMs). Our algorithm outperforms GMMs for data drawn from mixtures of gaussians in IR 2 and IR 6. Our algorithm is also automated with respect to param...
متن کامل